User devices now provide access to a wide variety of information. For example, digital image files, video and/or audio files, as well as resources for particular subjects or particular news articles, are accessible over the Internet. This information may be viewed in web browsers running on a user device and native applications running on a user device. Search engines crawl and index these resources to facilitate searching of the resources. Thus, for just about any subject in which a user may be interested, there is a large amount of information that is readily available to the user.
Many user devices are now also being used to consume streaming content, such as “on demand” video. For example, many users now watch television programs and other videos primarily by use of mobile devices or laptop computer and tablets. Additionally, many televisions that are now available have “smart” capabilities that enable the televisions to be used as web browsers and to run native applications that present content from particular content providers.
A user may desire information related to content that a user is viewing on a user device. Such information in this application is referred to as “contextual information,” as the information desired by the user is in the context of the active web page or media. Some user devices can provide an image of content that is displayed in an active window of a user device. A search system processes the image to identify entities and concepts and performs a search for those entities and concepts, and then provides one or more search results to the user device.